{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CI/CD for a KFP pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objectives:**\n",
    "1. Learn how to create a custom Cloud Build builder to pilote CAIP Pipelines\n",
    "1. Learn how to write a Cloud Build config file to build and push all the artifacts for a KFP\n",
    "1. Learn how to setup a Cloud Build Github trigger to rebuild the KFP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lab you will walk through authoring of a **Cloud Build** CI/CD workflow that automatically builds and deploys a KFP pipeline. You will also integrate your workflow with **GitHub** by setting up a trigger that starts the  workflow when a new tag is applied to the **GitHub** repo hosting the pipeline's code.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configuring environment settings\n",
    "\n",
    "Update  the `ENDPOINT` constat  with the settings reflecting your lab environment. \n",
    "\n",
    "Then endpoint to the AI Platform Pipelines instance can be found on the [AI Platform Pipelines](https://console.cloud.google.com/ai-platform/pipelines/clusters) page in the Google Cloud Console.\n",
    "\n",
    "1. Open the *SETTINGS* for your instance\n",
    "2. Use the value of the `host` variable in the *Connect to this Kubeflow Pipelines instance from a Python client via Kubeflow Pipelines SKD* section of the *SETTINGS* window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENDPOINT = '<YOUR_ENDPOINT>'\n",
    "PROJECT_ID = !(gcloud config get-value core/project)\n",
    "PROJECT_ID = PROJECT_ID[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the KFP CLI builder\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "In the cell below, write a docker file that\n",
    "* Uses `gcr.io/deeplearning-platform-release/base-cpu` as base image\n",
    "* Install the python package `kfp` with version `0.2.5`\n",
    "* Starts `/bin/bash` as entrypoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile kfp-cli/Dockerfile\n",
    "\n",
    "# TODO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build the image and push it to your project's **Container Registry**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IMAGE_NAME='kfp-cli'\n",
    "TAG='latest'\n",
    "IMAGE_URI='gcr.io/{}/{}:{}'.format(PROJECT_ID, IMAGE_NAME, TAG)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exercise\n",
    "\n",
    "In the cell below, use `gcloud builds` to build the `kfp-cli` Docker image and push it to the project gcr.io registry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud builds # COMPLETE THE COMMAND"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the **Cloud Build** workflow.\n",
    "\n",
    "### Exercise\n",
    "\n",
    "In the cell below, you'll complete the `cloudbuild.yaml` file describing the CI/CD workflow and prescribing how environment specific settings are abstracted using **Cloud Build** variables.\n",
    "\n",
    "The CI/CD workflow automates the steps you walked through manually during `lab-02-kfp-pipeline`:\n",
    "1. Builds the trainer image\n",
    "1. Builds the base image for custom components\n",
    "1. Compiles the pipeline\n",
    "1. Uploads the pipeline to the KFP environment\n",
    "1. Pushes the trainer and base images to your project's **Container Registry**\n",
    "\n",
    "Although the KFP backend supports pipeline versioning, this feature has not been yet enable through the **KFP** CLI. As a temporary workaround, in the **Cloud Build** configuration a value of the `TAG_NAME` variable is appended to the name of the pipeline. \n",
    "\n",
    "The **Cloud Build** workflow configuration uses both standard and custom [Cloud Build builders](https://cloud.google.com/cloud-build/docs/cloud-builders). The custom builder encapsulates **KFP CLI**. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%writefile cloudbuild.yaml\n",
    "\n",
    "steps:\n",
    "# Build the trainer image\n",
    "- name: 'gcr.io/cloud-builders/docker'\n",
    "  args: ['build', '-t', 'gcr.io/$PROJECT_ID/$_TRAINER_IMAGE_NAME:$TAG_NAME', '.']\n",
    "  dir: $_PIPELINE_FOLDER/trainer_image\n",
    "  \n",
    "# TODO: Build the base image for lightweight components\n",
    "- name: # TODO\n",
    "  args: # TODO\n",
    "  dir: # TODO\n",
    "\n",
    "# Compile the pipeline\n",
    "# TODO: Set the environment variables below for the $_PIPELINE_DSL script\n",
    "# HINT: https://cloud.google.com/cloud-build/docs/configuring-builds/substitute-variable-values\n",
    "- name: 'gcr.io/$PROJECT_ID/kfp-cli'\n",
    "  args:\n",
    "  - '-c'\n",
    "  - |\n",
    "    dsl-compile --py $_PIPELINE_DSL --output $_PIPELINE_PACKAGE\n",
    "  env:\n",
    "  - 'BASE_IMAGE= # TODO\n",
    "  - 'TRAINER_IMAGE= # TODO\n",
    "  - 'RUNTIME_VERSION= # TODO\n",
    "  - 'PYTHON_VERSION= # TODO\n",
    "  - 'COMPONENT_URL_SEARCH_PREFIX= # TODO\n",
    "  - 'USE_KFP_SA=$_USE_KFP_SA'\n",
    "  dir: $_PIPELINE_FOLDER/pipeline\n",
    "  \n",
    "# Upload the pipeline\n",
    "# TODO: Use the kfp-cli Cloud Builder and write the command to upload the ktf pipeline \n",
    "- name: # TODO\n",
    "  args:\n",
    "  - '-c'\n",
    "  - |\n",
    "    # TODO\n",
    "  dir: $_PIPELINE_FOLDER/pipeline\n",
    "\n",
    "\n",
    "# Push the images to Container Registry\n",
    "# TODO: List the images to be pushed to the project Docker registry\n",
    "images: # TODO\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manually triggering CI/CD runs\n",
    "\n",
    "You can manually trigger **Cloud Build** runs using the `gcloud builds submit` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SUBSTITUTIONS=\"\"\"\n",
    "_ENDPOINT={},\\\n",
    "_TRAINER_IMAGE_NAME=trainer_image,\\\n",
    "_BASE_IMAGE_NAME=base_image,\\\n",
    "TAG_NAME=test,\\\n",
    "_PIPELINE_FOLDER=.,\\\n",
    "_PIPELINE_DSL=covertype_training_pipeline.py,\\\n",
    "_PIPELINE_PACKAGE=covertype_training_pipeline.yaml,\\\n",
    "_PIPELINE_NAME=covertype_continuous_training,\\\n",
    "_RUNTIME_VERSION=1.15,\\\n",
    "_PYTHON_VERSION=3.7,\\\n",
    "_USE_KFP_SA=True,\\\n",
    "_COMPONENT_URL_SEARCH_PREFIX=https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/\n",
    "\"\"\".format(ENDPOINT).strip()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud builds submit . --config cloudbuild.yaml --substitutions {SUBSTITUTIONS}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setting up GitHub integration\n",
    "\n",
    "### Exercise\n",
    "\n",
    "In this exercise you integrate your CI/CD workflow with **GitHub**, using [Cloud Build GitHub App](https://github.com/marketplace/google-cloud-build). \n",
    "You will set up a trigger that starts the CI/CD workflow when a new tag is applied to the **GitHub** repo managing the  pipeline source code. You will use a fork of this repo as your source GitHub repository."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 1: Create a fork of this repo\n",
    "[Follow the GitHub documentation](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) to fork this repo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Step 2: Create a **Cloud Build** trigger\n",
    "\n",
    "Connect the fork you created in the previous step to your Google Cloud project and create a trigger following the steps in the [Creating GitHub app trigger](https://cloud.google.com/cloud-build/docs/create-github-app-triggers) article. Use the following values on the **Edit trigger** form:\n",
    "\n",
    "|Field|Value|\n",
    "|-----|-----|\n",
    "|Name|[YOUR TRIGGER NAME]|\n",
    "|Description|[YOUR TRIGGER DESCRIPTION]|\n",
    "|Event| Tag|\n",
    "|Source| [YOUR FORK]|\n",
    "|Tag (regex)|.\\*|\n",
    "|Build Configuration|Cloud Build configuration file (yaml or json)|\n",
    "|Cloud Build configuration file location|/ workshops/kfp-caip-sklearn/lab-03-kfp-cicd/cloudbuild.yaml|\n",
    "\n",
    "\n",
    "Use the following values for the substitution variables:\n",
    "\n",
    "|Variable|Value|\n",
    "|--------|-----|\n",
    "|_BASE_IMAGE_NAME|base_image|\n",
    "|_COMPONENT_URL_SEARCH_PREFIX|https://raw.githubusercontent.com/kubeflow/pipelines/0.2.5/components/gcp/|\n",
    "|_ENDPOINT|[Your inverting proxy host]|\n",
    "|_PIPELINE_DSL|covertype_training_pipeline.py|\n",
    "|_PIPELINE_FOLDER|workshops/kfp-caip-sklearn/lab-03-kfp-cicd|\n",
    "|_PIPELINE_NAME|covertype_training_deployment|\n",
    "|_PIPELINE_PACKAGE|covertype_training_pipeline.yaml|\n",
    "|_PYTHON_VERSION|3.7|\n",
    "|_RUNTIME_VERSION|1.15|\n",
    "|_TRAINER_IMAGE_NAME|trainer_image|\n",
    "|_USE_KFP_SA|False|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Trigger the build\n",
    "\n",
    "To start an automated build [create a new release of the repo in GitHub](https://help.github.com/en/github/administering-a-repository/creating-releases). Alternatively, you can start the build by applying a tag using `git`. \n",
    "```\n",
    "git tag [TAG NAME]\n",
    "git push origin --tags\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
